Toward meaningful test collections for information integration benchmarking

Meaningful comparison of algorithms for search, extraction and aggregation across heterogeneous sources will require welldesigned benchmarks, preferably based on freely available test collections. In this paper, we discuss issues which will inevitably arise during the construction of such benchmarks. We argue that the creation of benchmarks requires careful consideration of the evaluation methods. These go hand-in-hand with the selection of an appropriate test collection. Conversely, selection of inappropriate tasks or test collections can lead to false or incomplete conclusions about the effectiveness of a retrieval system which addresses information integration challenges. We draw on our experience with the development of collections for the evaluation of information retrieval systems focused on Web data and tasks. Our particular interest is in effective retrieval over heterogeneous data sources.